Container build process

The source code was taken from: https://github.com/paulbouwer/hello-kubernetes

Repository with images: https://quay.io/repository/petr_ruzicka/myc-hello-kubernetes?tab=tags

Requirements

Generic

Multiple Dockerfile files / sources in one git repository
Multi-arch container images for each Dockerfile
Image should be signed (cosign)
SBOM (Software Bill Of Materials)
SLSA Provenance
Build process should never overwrite existing tags (1.2.3) in container repository (doesn't apply for "latest" or "br-")
Container image should be scanned for "Critical" vulnerabilities before it is pushed to container registry
There should be a way to push container to registry also with CRITICAL vulnerabilities (how to progress when there are CRITICAL vulnerabilities which can not be fixed "immediately")
- Set container_image_vulnerability_checks=false

Tag specific

I want to build test container for specific PR / branch with expiration date
I want to build only latest tag manually (form main branch)
Newly released images must create/update following tags in container repository (example):
- 1
- 1.2
- 1.2.1
- latest

Local tests

Build commands:

docker build -f ./src/app/Dockerfile -t myc-hello-kubernetes:latest src/app
docker build -f ./src/app/Dockerfile-node-18-alpine -t myc-hello-kubernetes:latest-alpine src/app
docker build -f ./src/app/Dockerfile-node-18-debian-slim -t myc-hello-kubernetes:latest-debian-slim src/app
docker build -f ./src/app/Dockerfile-nodejs18-distroless -t myc-hello-kubernetes:latest-distroless src/app
docker build -f ./src/app/Dockerfile-nodejs-16-ubi -t myc-hello-kubernetes:latest-ubi src/app

Run commands:

docker run -p 8080:8080 --rm myc-hello-kubernetes:latest
docker run -p 8080:8080 --rm myc-hello-kubernetes:latest-alpine
docker run -p 8080:8080 --rm myc-hello-kubernetes:latest-debian-slim
docker run -p 8080:8080 --rm myc-hello-kubernetes:latest-distroless
docker run -p 8080:8080 --rm myc-hello-kubernetes:latest-ubi

curl http://localhost:8080/

Debug container:

docker run -it --rm --entrypoint=/bin/sh --user root -p 8080:8080 myc-hello-kubernetes:latest

Run in Kubernetes:

kubectl run myc-hello-kubernetes --image=quay.io/petr_ruzicka/myc-hello-kubernetes

Cosign - verify signatures / SBOMs

Example repositories:

Notes:

Test commands:

IMAGE="bridgecrew/checkov"
IMAGE="quay.io/petr_ruzicka/malware-cryptominer-container:2.0.0"
IMAGE="ghcr.io/external-secrets/external-secrets:v0.7.1"

docker buildx imagetools inspect "${IMAGE}"
cosign verify "${IMAGE}" | jq
cosign verify "${IMAGE}" | jq -r '.[].optional| .Issuer + " | " + .Subject + " | " + .githubWorkflowRef + " | https://rekor.tlog.dev/?logIndex=" + (.Bundle.Payload.logIndex|tostring)'
cosign triangulate "${IMAGE}"
cosign tree "${IMAGE}"
cosign verify-attestation --type slsaprovenance "${IMAGE}"
cosign verify-attestation --type slsaprovenance "${IMAGE}" | jq '.payload |= @base64d | .payload | fromjson'
cosign verify-attestation --type cyclonedx "${IMAGE}"
cosign verify-attestation --type cyclonedx "${IMAGE}" | jq '.payload |= @base64d | .payload | fromjson'
trivy image --debug --sbom-sources rekor "${IMAGE}"
cosign verify-attestation --type cyclonedx "${IMAGE}" | jq -r '.payload' | base64 -d | jq -r '.predicate.Data' > /tmp/sbom.cdx.json
trivy sbom /tmp/sbom.cdx.json
trivy image --scanners license "${IMAGE}"
trivy image --compliance docker-cis "${IMAGE}"
trivy image --scanners none --image-config-scanners config "${IMAGE}"
trivy image --scanners none --image-config-scanners secret "${IMAGE}"

Test commands with other images:

IMAGE="registry.k8s.io/kube-apiserver-amd64:v1.24.0"
IMAGE="quay.io/petr_ruzicka/malware-cryptominer-container:2.0.0"
IMAGE="quay.io/cilium/cilium:v1.13.0"
IMAGE="quay.io/metallb/controller:latest"
IMAGE="quay.io/costoolkit/elemental-operator:v1.1.0"
IMAGE="ghcr.io/fluxcd/source-controller:v0.34.0"
IMAGE="bridgecrew/checkov"

skopeo inspect --raw "docker://${IMAGE}"
cosign verify "${IMAGE}" | jq
cosign verify "${IMAGE}" | jq -r '.[].optional| .Issuer + "-" + .Subject'
cosign triangulate "${IMAGE}"
cosign tree "${IMAGE}"

rekor-cli get --uuid 68a53d0e75463d805dc9437dda5815171502475dd704459a5ce3078edba96226 --format json | jq -r .Attestation | base64 --decode | jq
curl -s https://rekor.sigstore.dev/api/v1/log/entries/68a53d0e75463d805dc9437dda5815171502475dd704459a5ce3078edba96226 | jq

rekor-cli search --email petr.ruzicka@gmail.com
# https://rekor.tlog.dev/?email=petr.ruzicka@gmail.com

rekor-cli get --log-index 8757761

docker manifest inspect "${IMAGE}"
cosign download sbom --platform=linux/amd64 ghcr.io/google/ko
cosign download sbom --platform=linux/amd64 cgr.dev/chainguard/node
cosign download sbom "${IMAGE}"

# Image Manifest
curl -s -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' "https://quay.io/v2/jetstack/cert-manager-controller/manifests/v1.9.1" | jq
# Manifest List
curl -s -H 'Accept: application/vnd.docker.distribution.manifest.list.v2+json' "https://quay.io/v2/jetstack/cert-manager-controller/manifests/v1.9.1" | jq
curl -s -H 'Accept: application/vnd.oci.image.index.v1+json' "https://quay.io/v2/jetstack/cert-manager-controller/manifests/v1.9.1" | jq

# Verify SBOM attestation
cosign verify-attestation --type cyclonedx bridgecrew/checkov | jq '.payload |= @base64d | .payload | fromjson | select(.predicateType == "https://cyclonedx.org/schema") | .predicate.Data'
cosign verify-attestation --type slsaprovenance --key https://ftp.suse.com/pub/projects/security/keys/container–key.pem registry.suse.com/bci/golang@sha256:35bc38ce40811b587a56bcfa328ef077c0703732e3bbedf4dbdf47f612cca04b | jq # DevSkim: ignore DS117838
cosign verify-attestation --type slsaprovenance ghcr.io/thomasvitale/band-service@sha256:388e8d292b55a7934bdaf11277ea9f33c3533258de92eb4b12085717dbdbd875 | jq '.payload |= @base64d | .payload | fromjson'

docker manifest inspect quay.io/jetstack/cert-manager-controller:v1.9.1

slsa-verifier verify-image "ghcr.io/chgl/kube-powertools@sha256:b76dc742957ae883f3ed0c4fe89b54d5c9a9de69a8bc531f9ee12ec995dab10d" --source-uri github.com/chgl/kube-powertools

# https://unifiedguru.com/highlights-from-the-buildkit-v0-11-release-docker/
docker buildx imagetools inspect moby/buildkit:latest --format '{{ json .Provenance }}'
docker buildx imagetools inspect moby/buildkit:latest --format '{{ json (index .Provenance "linux/amd64").SLSA.invocation.configSource }}'
docker buildx imagetools inspect moby/buildkit:latest --format '{{ (index .Provenance "linux/amd64").SLSA.builder.id }}'
docker buildx imagetools inspect moby/buildkit:latest --format '{{ range (index .SBOM "linux/amd64").SPDX.packages }}{{ println .name }}{{ end }}'

docker buildx imagetools inspect ghcr.io/fluxcd/source-controller:v0.34.0 --format "{{ json (index .Provenance \"linux/amd64\").SLSA}}"
docker buildx imagetools inspect ghcr.io/fluxcd/source-controller:v0.34.0 --format "{{ json (index .SBOM \"linux/amd64\").SPDX}}"
docker sbom fluxcd/source-controller:v0.34.0

# https://registry-ui.chainguard.app/?image=quay.io/petr_ruzicka/malware-cryptominer-container:1
# https://registry-ui.chainguard.app/?image=cgr.dev/chainguard/nginx:latest

cosign verify --certificate-github-workflow-repository cilium/cilium --certificate-oidc-issuer https://token.actions.githubusercontent.com --certificate-github-workflow-name "Image Release Build" --certificate-github-workflow-ref refs/tags/v1.13.0 quay.io/cilium/cilium:v1.13.0 | jq
cosign verify --certificate-github-workflow-repository cilium/cilium --certificate-oidc-issuer https://token.actions.githubusercontent.com --attachment sbom quay.io/cilium/cilium:v1.13.0 | jq

Container scanners

I was looking for the vulnerability which affects only the specific architecture (for example: arm):

https://security.snyk.io/vuln/SNYK-DEBIAN10-GLIBC-564233

trivy image --severity HIGH,CRITICAL --platform=linux/amd64 node:18.10.0-buster-slim | grep CVE-2020-6096
trivy image --severity HIGH,CRITICAL --platform=linux/arm64 node:18.10.0-buster-slim | grep CVE-2020-6096

Possible build examples

Build container images with tag latest form main branch and ignore vulnerability scanners

gh workflow run container-build.yml -f container_registry_push=true -f container_image_expires_after=30 -f container_image_skip_vulnerability_checks=true
sleep 100

Tag "main" branch

TAG="8.0.55"

git tag "v${TAG}-beta.0" && git push origin "v${TAG}-beta.0"
sleep 10
WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId')
gh run watch "${WORKLOAD_ID}"
# In case some containers has security vulnerabilities they will not be pushed to Container Registry by default
# If you want to do a force push please use something like:
gh workflow run container-build.yml --ref="v${TAG}-beta.0" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true
sleep 100

git tag "v${TAG}-beta.1" && git push origin "v${TAG}-beta.1"
sleep 10
WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId')
gh run watch "${WORKLOAD_ID}"
gh workflow run container-build.yml --ref="v${TAG}-beta.1" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true
sleep 100

git tag "v${TAG}-rc.0" && git push origin "v${TAG}-rc.0"
sleep 10
WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId')
gh run watch "${WORKLOAD_ID}"
gh workflow run container-build.yml --ref="v${TAG}-rc.0" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true
sleep 100

git tag "v${TAG}-rc.1" && git push origin "v${TAG}-rc.1"
sleep 10
WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId')
gh run watch "${WORKLOAD_ID}"
gh workflow run container-build.yml --ref="v${TAG}-rc.1" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true
sleep 100

git tag "v${TAG}" && git push origin "v${TAG}"
sleep 10
WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId')
gh run watch "${WORKLOAD_ID}"
gh workflow run container-build.yml --ref="v${TAG}" -f container_registry_push=true -f container_image_expires_after=365 -f container_image_skip_vulnerability_checks=true

Run build process form "main" branch - expires in 300 days

gh workflow run container-build.yml -f container_registry_push=true -f container_image_expires_after=300 -f container_image_skip_vulnerability_checks=true
sleep 10
WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId')
gh run watch "${WORKLOAD_ID}"

Run build process form "fix" branch - expires in 30 days (manual workflow execution is needed)

git checkout -b fix && git push
gh workflow run container-build.yml --ref=fix -f container_registry_push=true -f container_image_expires_after=30 -f container_image_skip_vulnerability_checks=true
sleep 10
WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId')
gh run watch "${WORKLOAD_ID}"
git checkout main && git branch -d fix && git push origin -d fix

Run build process form "fix2" branch form PR - expires in 30 days (manual workflow execution is needed)

git checkout -b fix2
date > date.txt
git add date.txt && git commit -m "fix(container-build): date" && git push
PR_URL=$(gh pr create --fill)
gh workflow run container-build.yml --ref=fix2 -f container_registry_push=true -f container_image_expires_after=30 -f container_image_skip_vulnerability_checks=true
sleep 10
WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId')
gh run watch "${WORKLOAD_ID}"
date > date.txt
git add date.txt && git commit -m "fix(container-build): date2" && git push
gh workflow run container-build.yml --ref=fix2 -f container_registry_push=true -f container_image_expires_after=30 -f container_image_skip_vulnerability_checks=true
sleep 10
WORKLOAD_ID=$(gh run list --workflow=container-build.yml --limit 1 --json databaseId | jq -r '.[].databaseId')
gh run watch "${WORKLOAD_ID}"
gh pr close --delete-branch "${PR_URL}"

Scheduled build - expires in 300 days

Name		Name	Last commit message	Last commit date
Latest commit History 185 Commits
.github		.github
src/app		src/app
.checkov.yml		.checkov.yml
.gitignore		.gitignore
.markdownlint.yml		.markdownlint.yml
.mega-linter.yml		.mega-linter.yml
.stylelintrc.json		.stylelintrc.json
LICENSE		LICENSE
README.md		README.md
lychee.toml		lychee.toml
package.json		package.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.github

.github

src/app

src/app

.checkov.yml

.checkov.yml

.gitignore

.gitignore

.markdownlint.yml

.markdownlint.yml

.mega-linter.yml

.mega-linter.yml

.stylelintrc.json

.stylelintrc.json

LICENSE

LICENSE

README.md

README.md

lychee.toml

lychee.toml

package.json

package.json

Repository files navigation

Container build process

Requirements

Generic

Tag specific

Local tests

Cosign - verify signatures / SBOMs

Container scanners

Possible build examples

About

Releases

Packages 1

Contributors 3

Languages

License

ruzickap/container-build

Folders and files

Latest commit

History

Repository files navigation

Container build process

Requirements

Generic

Tag specific

Local tests

Cosign - verify signatures / SBOMs

Container scanners

Possible build examples

About

Topics

Resources

License

Stars

Watchers

Forks

Languages